Visualization and animation of realistic 3D computer graphics models are important for many applications, such as computer games, movies, advertisement, and virtual environments. Generally, the following methods are used to reproduce a visual appearance of real objects: model-based rendering, image-based rendering, and hybrid rendering that uses both models and images.
In model-based methods, the reflectance properties of 3D objects are typically acquired by 3D photography. For explicit appearance models, a parametric bi-directional reflectance distribution function (BRDF) is fit to the acquired image data. Parametric BRDFs can be rendered efficiently on graphics hardware. The underlying model can be animated and deformed using well-developed techniques, such vertex blending. However, parametric BRDFs cannot acquire many effects of real-world materials, such as translucency, inter-reflections, self-shadowing, and subsurface scattering.
On the other hand, image-based rendering is well suited to acquire and represent objects having a complex appearance. However, image-based rendering is restricted to viewpoints of the acquired images. The lack of a 3D model makes deformations and animations very difficult, if not impossible.
Consequently, hybrid methods have become very popular. Hybrid methods parameterize acquired images onto a 3D model. The 3D model is then used for new viewpoints and animation. This representation is commonly known as a surface light field.
However, surface light fields can only provide novel renderings with a fixed lighting that corresponds to the acquired images. That is a severe limitation when it is desired to render the object in a new environment or under dynamically changing lighting.
A better hybrid representation uses a surface reflectance field. The surface reflectance field expresses an appearance of an object for many possible lighting configurations. Objects with arbitrary reflectance properties can be rendered from any arbitrary viewpoint under new lighting.
However, up to now the deformation and animation of objects with reflectance fields has not been possible.
In general, animating an image-based or hybrid appearance representation requires a process to evaluate the image-based data to simulate varying object deformations.
It is a problem to preserve the visual appearance of the deformed object surface. In other words, the perceptual impression of material properties of the object under mutable lighting conditions needs to be preserved as the object is deformed.
Therefore, there is a need to animate an object with a surface reflectance field with arbitrary deformations.
The prior art for three-dimensional rendering generally describes methods that acquire object shape and appearance from images. Some 3D rendering methods fit a parametric BRDF model to the acquired appearance data. However, parametric BRDFs cannot represent many of the reflectance properties of real objects.
In contrast, image-based appearance representations make no assumptions about the reflection property of materials. A lumigraph combines a 3D model, e.g., a visual hull of the object, with an image-based representation. Images of the object are represented as a dense light-field, and the model is used to improve light-field interpolation, i.e., to minimize ghosting.
To reduce the amount of image data required, view-dependent texture mapping can be used. Texture mapping uses a simple model and sparse texture data. That method is effective despite an approximate 3D model. However, that method has some limitations for highly specular surfaces due to a relatively small number of textures.
Surface light fields are a more general and efficient representation because surface light fields parameterize the image data onto the model surface. Surface light fields can either be associated with an accurate, high-density model, or with coarse triangular meshes for objects with low geometric complexity.
Some methods compress the light field data such that the models can be rendered in real-time using graphics hardware. To improve the appearance of complex object silhouettes, surface light fields can be combined with view-dependent opacity data into opacity light fields.
Unstructured lumigraph rendering is an effective method for rendering both surface and opacity light fields. Although surface light fields are capable of reproducing important global effects such as inter-reflections and self-shadowing, those methods only show the object under fixed lighting.
To overcome that limitation, surface reflectance fields have been used. The surface reflectance field of an object represents radiant light reflected from a surface under many possible incident lighting conditions. In practice, the surface reflectance field is sampled sparsely, and the sparse samples are interpolated while rendering. To further reduce the amount of data, most prior art reflectance fields are acquired for a single view. For approximately planar model geometry and diffuse surfaces, the data can be compressed further by fitting a parametric function to the surface reflectance field.
Surface reflectance fields can be generated from data acquired by 3D photography, W. Matusik, H. Pfister, A. Ngan, P. Beardsley, R. Ziegler, and L. McMillan, “Image-based 3D photography using opacity hulls,” ACM Transaction on Graphics, 21(3): pp. 427–437, July 2002. That system acquires surface reflectance fields for over four hundred views using multiple cameras, turntables, and a rotating array of lights. The approximate geometry is a visual hull of the objects, U.S. patent application, 20020159628, Matusik et al., filed Oct. 31, 2002, “Image-based 3D digitizer.”
Although it is an important aspect for many practical applications, the animation of image-based data has received very little attention. Wood et al. describe arbitrary deformations on a surface light field and produce plausible renderings of the deformed model, D. Wood, D. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. Salesin, and W. Stuetzle, “Surface light fields for 3d photography,” Computer Graphics, SIGGRAPH 2000 Proceedings, pages 287–296, July 2000. However, that method does not deal properly with the diffuse component of the surface color. That method only works for purely reflective isotropic BRDFs.
Feature-based light field morphing ‘blends’ two light fields into one based on the concept of ray-correspondencies, Z. Zhang, L. Wang, B. Guo, and H.-Y. Shum, “Feature-based light field morphing,” Computer Graphics, SIGGRAPH 2002 Proceedings, pages 457–464, July 2002. That method requires substantial user input to specify corresponding feature polygons between the two light fields. That method is not applicable to the general animation setting, and the method only works with static lighting.
Furukawa et al. describe a scanning system to acquire models of objects and spatially varying BRDFs, also called bi-directional texture functions (BTFs), R. Furukawa, H. Kawasaki, K. Ikeuchi, and M. Sakauchi, “Appearance based object modeling using texture database: Acquisition, compression and rendering,” Proceedings of the 13th Eurographics Workshop on Rendering, June 2002. That method uses tensor product expansion to compress the BTF data. That method relies on an exact model to support the BTF representation. Thus, the shape of the object has to be acquired with a range scanning device, which is expensive, time consuming, and complex. Moreover, that method does not explicitly address appearance preservation under non-uniform deformations.
Therefore, it is desired to provide a rendering method that can deform and animate models of objects using surface reflectance fields so that objects can be placed in virtual environments with arbitrary lighting, including dynamically changing lighting.